Route Hamming code sizes to the fastest kernel, and accept ragged sizes - #5520
Open
mnorris11 wants to merge 1 commit into
Open
Route Hamming code sizes to the fastest kernel, and accept ragged sizes#5520mnorris11 wants to merge 1 commit into
mnorris11 wants to merge 1 commit into
Conversation
Contributor
|
@mnorris11 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D115982541. |
meta-codesync
Bot
changed the title
mnorris11
pushed a commit
to mnorris11/faiss
that referenced
this pull request
Aug 14, 2026
Summary: `faiss::hammings()` used to `FAISS_THROW_IF_NOT(ncodes % 8 == 0)`, even though the other Hamming entry points (`hammings_knn_hc`, `hamming_range_search`) have always accepted code sizes that are not a whole number of 64-bit words. This lifts that restriction so `hammings()` matches the rest of the API. Ragged sizes are routed to a new out-of-line helper, `hammings_ragged()`, built on the existing `HammingComputer` family (which already carries a byte tail). Multiples of 8 keep taking the word-level kernels exactly as before, so there is no behavior change for existing callers. Also adds a `FAISS_NOINLINE` macro to `platform_macros.h` (MSVC `__declspec(noinline)` / GCC-Clang `__attribute__((noinline))`), used to keep `hammings_ragged()` out of line. ## Why FAISS_NOINLINE Letting the compiler inline `hammings_ragged()` into `hammings_fixSL()` instantiates the whole `HammingComputer` family into that function and roughly doubles its size (3,597 -> 7,565 bytes at the AVX2 level). That measurably slows the word-level code sizes that share the runtime-nwords loop. Benchmarked on a Xeon 8339HC (Cooper Lake, AVX-512), `mode/opt`, na=256 x nb=4096 = 1M pairs per call, one code size per process, A/B interleaved with order flipping, 300-400 samples per cell, replicated on two cores. Positive = inlining is slower, i.e. `noinline` wins. | ncodes | static AVX2 | dynamic dispatch (AVX512) | | --- | --- | --- | | 8, 16, 32, 64, 128 | ~0% | ~0% | | 24 | +3.4% | +53% | | 40 | +5.0% | +22% | | 48 | ~0% | +28% | | 56 | +4.4% | +35% | | 12, 20, 33 (ragged) | -5% | -5% to -12% | Two distinct mechanisms, both confirmed in the disassembly: - **Static AVX2** - the inlined version has higher register pressure and spills, adding exactly one reload per pair in the shared outer loop. Retired instruction counts (noise-free) show +1.00 instr/pair at ncodes 24/40/48/56 and 0.00 at 8/16/32/64/128. - **Dynamic dispatch** - the popcount inner loop is a byte-identical 24-byte sequence in both builds, but out of line it starts at `%32 == 0` (one 32-byte fetch window) and inlined at `%32 == 16` (straddles two). IPC drops 3.02 -> 2.36 at identical clock. This one is a code-placement effect: it is real today but is not something `noinline` reliably controls, and it could invert on a compiler upgrade. Note that the dedicated `ncodes` 8/16/32/64 kernels are unaffected either way - they have their own specialized `hammings_impl<nbits>` blocks that the ragged inline never touches. The trade: the ragged path itself is ~5% slower for being out of line. That is accepted here because the word-level sizes are the common case and are the ones that were already supported. Differential Revision: D115982541
mnorris11
force-pushed
the
export-D115982541
branch
from
August 14, 2026 21:59
aab66c6 to
8a491a2
Compare
meta-codesync
Bot
changed the title
mnorris11
force-pushed
the
export-D115982541
branch
from
August 15, 2026 03:31
8a491a2 to
4a52ad4
Compare
Summary: `faiss::hammings()` threw unless the code size was a whole number of 64-bit words, even though the other Hamming entry points accept ragged sizes. And for aligned sizes it always used the bit-level `hammings_impl<nbits>` kernels, never the hand-written `HammingComputer` family, which is faster at some sizes. Callers needing either case had to reach past `hammings()` into `with_simd_level_a0_spr` and `with_HammingComputer` and hand-roll the sweep plus a routing rule. Laser does this today in `KnnFaissBinaryIndex::measureAllCentroids` (D115840549) and can drop it. `hammings_fixSL()` now takes the `HammingComputer` path when the size is ragged, or when `prefer_hamming_computer()` says the computer wins there. That predicate is `constexpr` over `THE_SIMD_LEVEL`; sizes it does not name keep the kernels they use today. ## Which sizes Measured on Intel Cooper Lake (Xeon 8339HC) and AMD Genoa (EPYC, Zen 4), `mode/opt`, interleaved A/B, one code size per process, 250-600 samples per cell. Speedup from rerouting: | ncodes | AVX2 Intel | AVX2 AMD | AVX512 Intel | AVX512 AMD | | --- | --- | --- | --- | --- | | 16 | -25% | -3.0% | word kernel wins, not rerouted | same | | 32 | -25% | -12.6% | word kernel wins, not rerouted | see below | | 64 | -47% | -40.4% | **-83%** | **-63%** | ncodes 64 prefers the computer everywhere on x86: `hamming<512>` has no SIMD specialization above 256 bits, so it is a plain scalar popcount loop, and AVX-512 flags make it worse rather than better (15.0 -> 2.5 ns/pair Intel, 5.7 -> 2.1 AMD). ncodes 8 and the runtime-nwords sizes (24, 40, 48, 56, 128, 160) keep the word kernels, which beat the computer by 46-218%. ## Known regressions ncodes 10 and 33 lose 3-5% on AMD (10: +3.2% AVX2, +4.6% AVX512; 33: +4.3% AVX512). Both are ragged sizes whose routing is unchanged — shrinking the switch shifts the layout of the code around them. Reproducible at 600 samples on two cores. Everything else is within noise or faster. ## ncodes 32 at AVX512 differs by vendor Intel prefers the word kernel there by 27%; AMD prefers the computer by 23%. The table follows Intel, so AMD gives up ~23% at that one size. Fixing it requires a runtime predicate, which was measured and rejected: routing 32 on `SIMDConfig::avx512_split` gains 23% at ncodes 32 but costs 15-30% at ncodes 24/40/48/56, because `prefer_hamming_computer()` stops being `constexpr` and both paths must stay live. A compile-time Zen 4 SIMD level would capture it without that cost, but is a larger change than belongs here. ## Caveat on per-size numbers Sizes whose routing does not change still move by 5-15% between builds, both directions, both vendors: `hammings_fixSL()` is placement-sensitive and resizing its switch reshuffles it. ncodes 24 is the worst offender, swinging from -13.8% to +19.6% across builds differing only in unrelated code. Treat deltas outside the rerouted set as layout, not signal. Differential Revision: D115982541
mnorris11
force-pushed
the
export-D115982541
branch
from
August 15, 2026 03:40
4a52ad4 to
745ffb2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
faiss::hammings()threw unless the code size was a whole number of 64-bit words, even though the other Hamming entry points accept ragged sizes. And for aligned sizes it always used the bit-levelhammings_impl<nbits>kernels, never the hand-writtenHammingComputerfamily, which is faster at some sizes.Callers needing either case had to reach past
hammings()intowith_simd_level_a0_sprandwith_HammingComputerand hand-roll the sweep plus a routing rule. Laser does this today inKnnFaissBinaryIndex::measureAllCentroids(D115840549) and can drop it.hammings_fixSL()now takes theHammingComputerpath when the size is ragged, or whenprefer_hamming_computer()says the computer wins there. That predicate isconstexproverTHE_SIMD_LEVEL; sizes it does not name keep the kernels they use today.Which sizes
Measured on Intel Cooper Lake (Xeon 8339HC) and AMD Genoa (EPYC, Zen 4),
mode/opt, interleaved A/B, one code size per process, 250-600 samples per cell. Speedup from rerouting:ncodes 64 prefers the computer everywhere on x86:
hamming<512>has no SIMD specialization above 256 bits, so it is a plain scalar popcount loop, and AVX-512 flags make it worse rather than better (15.0 -> 2.5 ns/pair Intel, 5.7 -> 2.1 AMD).ncodes 8 and the runtime-nwords sizes (24, 40, 48, 56, 128, 160) keep the word kernels, which beat the computer by 46-218%.
Known regressions
ncodes 10 and 33 lose 3-5% on AMD (10: +3.2% AVX2, +4.6% AVX512; 33: +4.3% AVX512). Both are ragged sizes whose routing is unchanged — shrinking the switch shifts the layout of the code around them. Reproducible at 600 samples on two cores. Everything else is within noise or faster.
ncodes 32 at AVX512 differs by vendor
Intel prefers the word kernel there by 27%; AMD prefers the computer by 23%. The table follows Intel, so AMD gives up ~23% at that one size.
Fixing it requires a runtime predicate, which was measured and rejected: routing 32 on
SIMDConfig::avx512_splitgains 23% at ncodes 32 but costs 15-30% at ncodes 24/40/48/56, becauseprefer_hamming_computer()stops beingconstexprand both paths must stay live. A compile-time Zen 4 SIMD level would capture it without that cost, but is a larger change than belongs here.Caveat on per-size numbers
Sizes whose routing does not change still move by 5-15% between builds, both directions, both vendors:
hammings_fixSL()is placement-sensitive and resizing its switch reshuffles it. ncodes 24 is the worst offender, swinging from -13.8% to +19.6% across builds differing only in unrelated code. Treat deltas outside the rerouted set as layout, not signal.Differential Revision: D115982541